Reference chains for surviving live-heap samples (PROF-15341)#644
Reference chains for surviving live-heap samples (PROF-15341)#644jbachorik wants to merge 60 commits into
Conversation
…ALLOC_PERTURB_ for glibc test runs Antagonist races tracer-style context propagation against virtual-thread carrier churn to target the ContextStorageMode.THREAD stale-carrier use-after-free. Since such heap corruption manifests silently until an unrelated later allocation, also turn on glibc's own corruption checks by default for chaos runs and ddprof-test suites (skipped on musl and whenever a sanitizer/allocator already owns malloc via LD_PRELOAD). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
J9 hits a pre-existing, unrelated glibc heap-corruption bug that MALLOC_CHECK_ now surfaces as a hard abort; gate it off until that's investigated separately.
This comment has been minimized.
This comment has been minimized.
CI Test ResultsRun: #29515196421 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-16 16:42:51 UTC |
isRawPointer() only checked RAW_POINTER_MASK, letting raw J9 ASGCT BCIs that never went through encode() misroute into JVMSupport::resolve(), which asserts false for non-HotSpot VMs. Require ENCODED_MASK too.
It was registered in Main.java's antagonist factory but never added to either config's ANTAGONISTS string, so it silently never ran in CI.
Include ACTIVE_PHASE_MILLIS in STALE_RACER_SLEEP_MILLIS so racers wake as the next active phase begins instead of ~3s early. Bound stopGracefully's total wait to the caller's timeout via a shared deadline instead of giving each join/tryAcquire the full budget independently.
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125072101 |
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
…ntext via @trace driverLoop() called JavaProfiler.getInstance() directly, but JavaProfiler is a chaosCompileOnly dependency never bundled into chaos.jar and not visible to the app classloader (the agent's copy is shaded/relocated). The resulting NoClassDefFoundError is an Error, not an Exception, so it escaped the catch (Exception e) block and silently killed the driver thread — leaving activePhase stuck true and letting the sibling pinningChurnLoop run full-throttle for the rest of the test instead of its intended active/quiet duty cycle, which is the likely driver behind the OOM seen in reliability-chaos-aarch64 [profiler, gmalloc, 21.0.3-tem]. Rewrite the antagonist to only touch the tracer through @Trace-annotated methods (the same proven-safe dd-trace-api compileOnly pattern already used by TraceContextAntagonist), letting the real tracer and its virtual-thread instrumentation drive setContext/clearContext instead of calling com.datadoghq.profiler.* directly. Drops the custom-attribute/baggage simulation in favor of nested @trace hops plus forceUnmount and stale-buffer racer threads that still race OtelContextStorage's thread-scoped fallback across carrier churn. Verified with a 90s live run (JDK 21.0.11, profiler+tracer, gmalloc): no NoClassDefFoundError, no exceptions, completed cleanly with RC=0. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without a RUNTIME arg, chaos_check.sh silently proceeds with an empty value, which becomes --duration s in the chaos harness invocation and fails 15+ seconds later — after the JDK/agent/jar setup — with a NumberFormatException buried in the harness output, reported generically as "FAIL:Chaos harness crashed (RC=1)". Check for a missing RUNTIME right after arg parsing and fail immediately with a clear usage message. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Extract chaos_check.sh's logic into utils/run-chaos-harness.sh for local repro. Add a #ifdef DEBUG-only watchdog in ProfiledThread's free path (threadLocalData.cpp) that detects writes into freed OTel-context memory, and beef up VirtualThreadContextCascadeAntagonist's stale-carrier racing (continuous decoupled driver, randomized race window, dead-code removal). Fixes from review: Main.java antagonist-start try/finally, watcher thread signal-mask inheritance, racer-loop backoff under saturation, redundant frame_ut.cpp assertions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
watchFreedContextMemory() deliberately read freed ProfiledThread memory to detect stale writes, tripping ASan/TSan on every free (e.g. in forced_unwind_ut) unrelated to the actual race under investigation.
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
The skip's only justification was PROF-15360, closed as invalid in favor of PROF-15364 (isRawPointer() false-positive, fixed by this PR). No separate J9 heap-corruption issue is on record, so J9 now gets the same MALLOC_CHECK_/MALLOC_PERTURB_ coverage as glibc/HotSpot.
…profiler-only run
…IER mode Without --add-exports java.base/jdk.internal.misc=ALL-UNNAMED, OtelContextStorage silently falls back to ContextStorageMode.THREAD, which vthread-context-cascade was built to catch: virtual-thread context buffers pinned to a since-freed carrier's ProfiledThread, corrupting the heap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
profiler+tracer chaos runs OOMKilled on aarch64 (exit 137) with gmalloc/jemalloc but not tcmalloc, which already had aggressive-decommit tuning for the same thread-churn RSS-inflation issue. Apply the equivalent tuning to the other two allocators.
Measured on matching-arch hosts: this workload floats at ~2.6-2.75GB RSS regardless of MALLOC_CHECK_/allocator tuning, ruling that out as the OOM cause. Trim -Xmx to 1536m on aarch64 for real headroom, and log the container's actual cgroup memory limit so future OOMs are diagnosable from the job log directly.
Fixes a structural gap: a one-shot BFS search could finish walking the whole reachable graph before LivenessTracker's population-trend detection had enough epochs to flag a candidate, making later-created leaking objects permanently undiscoverable. - PainBudget (painBudget.h): leaky bucket over cost (ms), not event rate - gates how soon a restarted search may take its first pass. - ReferenceChainTracker::canAffordNewSearch()/restartSearch(): a terminal search restarts once LivenessTracker reports a leak candidate and the pain budget has drained; unaffected when generations=true isn't set. - FrontierTable::resetForRestart(), painbudget=N config sub-option (default 1%). - gtest coverage for PainBudget and the restart gate; verified against ExternalProcessReferenceChainTest (separate-process leaking-cache scenario), which now passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ce-chain JFR roundtrip test Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…shared test-JVM heap Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…owth _table_size was volatile but updated via __sync_bool_compare_and_swap, which has no synchronizes-with edge under the C++ memory model - switch to std::atomic<int> with relaxed ops. Also clamp the reference-chain tests' per-round allocation growth at round 10 (trend-eligibility point) so retry rounds stop adding unbounded garbage, which was causing OOME in CI's shared test-JVM heap.
The glibc-aarch64 job never installed gtest/gmock, unlike amd64 and musl jobs, so all native gtests silently skipped there. This surfaced as a hard failure once ReferenceChainJfrParserTest started depending on the roundtrip gtest actually producing its JFR file.
…from reference-chain BFS reconstruction Both mechanisms are probabilistic and hard to reliably co-occur in one JVM run; new JNI test seams (guarded by #ifdef DEBUG) let JUnit tests seed population history and tag a known live object directly, verifying each end-to-end in isolation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resetSearchStateForTest() and klassPopulationResetForTest() now stop the BFS thread / take _table_lock respectively before mutating state the thread otherwise owns exclusively. Also reset LivenessTracker's own population history before shouldReconstructReferrerChainThroughUnboundedCacheLeak() so a stale candidate from the prior test can't outrank CachedPayload, and align its per-round growth rate with the sibling test's.
StringDictionary::clearAll() (on profiler restart) wipes dict ids but JVMTI class tags survive, so ReferenceChainTracker::_class_tags held ids into a namespace that no longer existed. Add a generation counter to StringDictionary and have resolveLoadedClasses() detect a mismatch and force re-resolution, reusing existing JVMTI tags instead of always minting new ones. Also adds a firstpassbudget override for the root-seeded first BFS pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…samples cleanup_table()'s per-klass population accounting was gated on !forced, skipping it entirely whenever track()'s table-overflow branch triggered a forced sweep. Gate on a new is_epoch_owner check instead (true once per genuinely new GC epoch, forced or not), so a forced sweep still folds a sample using an already-cached klass id. resolveKlassId() (a real Class.getName() Java upcall) still only runs on the organic path - that part was a legitimate hot-path cost concern, just misdescribed as a JVMTI safety requirement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resetSearchStateForTest() only reset the frontier table's occupancy, not its capacity, so an earlier test's smaller framecap= permanently capped the singleton for every later test in the same JVM.
…ceChainTestSeamsTest An earlier test can leave the shared FrontierTable full/tiny (shouldReportAbandonedSearchOnTinyFrontierCap deliberately caps it at 1); without resetting first, tagAsReferenceChainRoot0()'s insert() fails deterministically.
Profiler::dump() discarded _jfr.dump()'s Error entirely, and OS::copyFile()'s sendfile() loop broke silently on short/failed copies - both are debug-only visibility gaps that would otherwise leave a lost reference-chain write or truncated chunk copy unexplained in CI logs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same debug-only seeded-representative short-circuit already used by ReferenceChainTrackingTest, ported to the external-process scenario: propagates ddprof_test.config to the child JVM so it can bypass LivenessTracker's probabilistic slope detection instead of depending on it to notice CachedPayload within the fixed round/timeout budget. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ubuntu.com Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1299316 to
baba070
Compare
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
… runs ObjectSampler::start()/stop() gated calls into LivenessTracker on its own (freshly correct) flags, but LivenessTracker never got a chance to refresh its own _gc_generations from a run that requested neither liveness nor generations tracking - leaving it stuck at whatever the previous test in the same JVM last set it to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
…GcRoot The one-shot root-seeded BFS walk can fire before round 1's addAll() in this shared, no-forkEvery JVM, freezing gcRootHolder's children as permanently empty. Seed one live ChainLink before resetting the search so the backing array is never empty when the walk snapshots it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341): a JVMTI
FollowReferences-basedBFS walk (
ReferenceChainTracker) tags reachable objects and recordsreferrer-type chains back toward a GC root, bridged to
LivenessTracker::selectLeakCandidates()'s population-slope leak detection.This revision also closes a structural gap found while adding end-to-end
coverage: a one-shot search could finish walking the whole reachable graph
before population-trend detection had accumulated enough GC epochs to flag a
candidate, making any object allocated after that point permanently
undiscoverable. Search restarts are now gated on an actual leak indication
existing (so a restarted walk is very likely to still find the object) plus
a
PainBudgetcooldown - a leaky bucket over cumulative safepoint timespent, not a fixed interval, so a cheap search can restart sooner than an
expensive one.
Motivation:
PROF-15341 - give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive".
Additional Notes:
See
doc/architecture/LiveHeapReferenceChains.mdanddoc/architecture/LiveHeapReferenceChains-RemainingWorkPlan.mdfor the fulldesign history, open questions, and what's still provisional (per-pass
budget/hop-cap/TTL/frontier-cap/pain-budget-refill-rate defaults are all
round, unbenchmarked placeholders pending a future measurement pass).
How to test the change?:
./gradlew :ddprof-lib:gtestDebug- full native unit suite, including newPainBudgetTest/SearchRestartTestcoverage for the restart gate../gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"-genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM and asserts the reconstructed chain's leaf class.
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"-in-process coverage for the BFS engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code